| Statistical Mechanics of Linear and Nonlinear Time-Domain Ensemble Learning - Seiji Miyoshi Conventional ensemble learning combines students in the space domain. In this paper, however, we combine students in the time domain and call it time-domain ensemble learning. We analyze, compare, and discuss the generalization performances regarding time-domain ensemble learning of both a linear model and a nonlinear model. Analyzing in the framework of online learning using a statistical mechanical method, we show the qualitatively different behaviors between the two models... Downloads: 15 | |

| One step RSB scheme for the rate distortion function - Tatsuto Murayama We apply statistical mechanics to an inverse problem of linear mapping to investigate the physics of the irreversible compression. We use the replica symmetry breaking (RSB) technique with a toy model to demonstrate the Shannon's result. The rate distortion function, which is widely known as the theoretical limit of the compression with a fidelity criterion, is derived using the Parisi one step RSB scheme... Downloads: 1 | |

| Acceleration effect caused by the Onsager reaction term in a frustrated coupled oscillator system - Toru Aonishi The role of the Onsager reaction term (ORT) is not yet well understood in frustrated coupled oscillator systems, since the Thouless-Anderson-Palmer (TAP) and replica methods cannot be directly applied to these non-equilibrium systems. In this paper, we consider two oscillator associative memory models, one with symmetric and one with asymmetric dilution of coupling. These two systems are ideal for evaluating the effect of the ORT, because, with the exception of the ORT, they have the same order ... Downloads: 2 | |

| Effect of Slow Switching in On-line Learning for Ensemble Teachers - Seiji Miyoshi We have analyzed the generalization performance of a student which slowly switches ensemble teachers. By calculating the generalization error analytically using statistical mechanics in the framework of on-line learning, we show that the dynamical behaviors of generalization error have the periodicity that is synchronized with the switching period and the behaviors differ with the number of ensemble teachers... Downloads: 2 | |

| Pulse-coupled resonate-and-fire models - Keiji Miura We analyze two pulse-coupled resonate-and-fire neurons. Numerical simulation reveals that an anti-phase state is an attractor of this model. We can analytically explain the stability of anti-phase states by means of a return map of firing times, which we propose in this paper. The resultant stability condition turns out to be quite simple. The phase diagram based on our theory shows that there are two types of anti-phase states... Downloads: 2 | |

| Dynamically-Coupled Oscillators -- Cooperative Behavior via Dynamical Interaction -- - Toru Aonishi We propose a theoretical framework to study the cooperative behavior of dynamically coupled oscillators (DCOs) that possess dynamical interactions. Then, to understand synchronization phenomena in networks of interneurons which possess inhibitory interactions, we propose a DCO model with dynamics of interactions that tend to cause 180-degree phase lags. Employing an approach developed here, we demonstrate that although our model displays synchronization at high frequencies, it does not exhibit s... Downloads: 2 | |

| Mixed state on a sparsely encoded associative memory model - Tomoyuki Kimoto In the present paper, we analyze symmetric mixed states corresponding to the so-called concept formation on a sparsely encoded associative memory model with 0-1 neurons. Three types of mixed states, OR, AND and a majority decision mixed state are described as typical examples. Each element of the OR mixed state is composed of corresponding memory pattern elements by means of the OR-operation. The other two types are similarly defined... Downloads: 2 | |

| Statistical mechanics of lossy compression using multilayer perceptrons - Kazushi Mimura Statistical mechanics is applied to lossy compression using multilayer perceptrons for unbiased Boolean messages. We utilize a tree-like committee machine (committee tree) and tree-like parity machine (parity tree) whose transfer functions are monotonic. For compression using committee tree, a lower bound of achievable distortion becomes small as the number of hidden units K increases. However, it cannot reach the Shannon bound even where K -> infty... Downloads: 2 | |

| Ensemble learning of linear perceptron; Online learning theory - Kazuyuki Hara Within the framework of on-line learning, we study the generalization error of an ensemble learning machine learning from a linear teacher perceptron. The generalization error achieved by an ensemble of linear perceptrons having homogeneous or inhomogeneous initial weight vectors is precisely calculated at the thermodynamic limit of a large number of input elements and shows rich behavior. Our main findings are as follows... Downloads: 3 | |

| Generating functional analysis of CDMA detection dynamics - Kazushi Mimura We investigate the detection dynamics of the parallel interference canceller (PIC) for code-division multiple-access (CDMA) multiuser detection, applied to a randomly spread, fully syncronous base-band uncoded CDMA channel model with additive white Gaussian noise (AWGN) under perfect power control in the large-system limit. It is known that the predictions of the density evolution (DE) can fairly explain the detection dynamics only in the case where the detection dynamics converge... Downloads: 6 | |

| Statistical Mechanics of Online Learning for Ensemble Teachers - Seiji Miyoshi We analyze the generalization performance of a student in a model composed of linear perceptrons: a true teacher, ensemble teachers, and the student. Calculating the generalization error of the student analytically using statistical mechanics in the framework of on-line learning, it is proven that when learning rate $\eta 1$, the properties are completely reversed. If the variety of the ensemble teachers is rich enough, the direction cosine between the true teacher and the student becomes unity ... | |

| Stochastic transitions of attractors in associative memory models with correlated noise - Masaki Kawamura We investigate dynamics of recurrent neural networks with correlated noise to analyze the noise's effect. The mechanism of correlated firing has been analyzed in various models, but its functional roles have not been discussed in sufficient detail. Aoyagi and Aoki have shown that the state transition of a network is invoked by synchronous spikes. We introduce two types of noise to each neuron: thermal independent noise and correlated noise... | |

| Statistical Mechanics of the Bayesian Image Restoration under Spatially Correlated Noise - Jun Tsuzurugi We investigated the use of the Bayesian inference to restore noise-degraded images under conditions of spatially correlated noise. The generative statistical models used for the original image and the noise were assumed to obey multi-dimensional Gaussian distributions whose covariance matrices are translational invariant. We derived an exact description to be used as the expectation for the restored image by the Fourier transformation and restored an image distorted by spatially correlated noise... | |

| Residual Energies after Slow Quantum Annealing - Sei Suzuki Features of the residual energy after the quantum annealing are investigated. The quantum annealing method exploits quantum fluctuations to search the ground state of classical disordered Hamiltonian. If the quantum fluctuation is reduced sufficiently slowly and linearly by the time, the residual energy after the quantum annealing falls as the inverse square of the annealing time. We show this feature of the residual energy by numerical calculations for small-sized systems and derive it on the b... | |

| On-line learning through simple perceptron with a margin - Kazuyuki Hara We analyze a learning method that uses a margin $\kappa$ {\it a la} Gardner for simple perceptron learning. This method corresponds to the perceptron learning when $\kappa=0$, and to the Hebbian learning when $\kappa \to \infty$. Nevertheless, we found that the generalization ability of the method was superior to that of the perceptron and the Hebbian methods at an early stage of learning. We analyzed the asymptotic property of the learning curve of this method through computer simulation and fo... | |

| Analysis of on-line learning when a moving teacher goes around a true teacher - Seiji Miyoshi In the framework of on-line learning, a learning machine might move around a teacher due to the differences in structures or output functions between the teacher and the learning machine or due to noises. The generalization performance of a new student supervised by a moving machine has been analyzed. A model composed of a true teacher, a moving teacher and a student that are all linear perceptrons with noises has been treated analytically using statistical mechanics... | |

| Storage Capacity Diverges with Synaptic Efficiency in an Associative Memory Model with Synaptic Delay and Pruning - Seiji Miyoshi It is known that storage capacity per synapse increases by synaptic pruning in the case of a correlation-type associative memory model. However, the storage capacity of the entire network then decreases. To overcome this difficulty, we propose decreasing the connecting rate while keeping the total number of synapses constant by introducing delayed synapses. In this paper, a discrete synchronous-type model with both delayed synapses and their prunings is discussed as a concrete example of the pro... | |

| Correlation of Firing in Layered Associative Neural Networks - Michiko Yamana There is growing interest in a phenomenon called the ``synfire chain'', in which firings of neurons propagate from pool to pool in the chain. The mechanism of the synfire chain has been analyzed by many resarchers. Keeping the synfire chain phenomenon in mind, we investigate a layered associative memory neural network model, in which patterns are embedded in connections between neurons. In this model, we also include uniform noise in connections, which induces common input in the next layer... | |

| Mixed states on neural network with structural learning - Tomoyuki Kimoto We investigated the properties of mixed states in a sparsely encoded associative memory model with a structural learning method. When mixed states are made of s memory patterns, s types of mixed states, which become equilibrium states of the model, can be generated. To investigate the properties of s types of the mixed states, we analyzed them using the statistical mechanical method. We found that the storage capacity of the memory pattern and the storage capacity of only a particular mixed stat... | |

| Multibranch entrainment and slow evolution among branches in coupled oscillators - Toru Aonishi In globally coupled oscillators, it is believed that strong higher harmonics of coupling functions are essential for multibranch entrainment (MBE), in which there exist many stable states, whose number scales as $\sim$ $O(\exp N)$ (where N is the system size). The existence of MBE implies the non-ergodicity of the system. Then, because this apparent breaking of ergodicity is caused by microscopic energy barriers, this seems to be in conflict with a basic principle of statistical physics... Downloads: 3 | |

| Generating Functional Analysis for Iterative CDMA Multiuser Detectors - Kazushi Mimura We investigate the detection dynamics of a soft parallel interference canceller (soft-PIC), which includes a hard-PIC as a special case, for code-division multiple-access (CDMA) multiuser detection, applied to a randomly spread, fully synchronous base-band uncoded CDMA channel model with additive white Gaussian noise under perfect power control in the large-system limit. We analyze the detection dynamics of some iterative detectors, namely soft-PIC, the Onsager-reaction-cancelling parallel inter... Downloads: 4 | |

| Transient dynamics for sequence processing neural networks - Masaki Kawamura An exact solution of the transient dynamics for a sequential associative memory model is discussed through both the path-integral method and the statistical neurodynamics. Although the path-integral method has the ability to give an exact solution of the transient dynamics, only stationary properties have been discussed for the sequential associative memory. We have succeeded in deriving an exact macroscopic description of the transient dynamics by analyzing the correlation of crosstalk noise... Downloads: 3 | |

| Robustness of retrieval properties against imbalance between long-term potentiation and depression of spike-timing-dependent plasticity - Narihisa Matsumoto Spike-timing-dependent plasticity (STDP) has recently been shown in some physiological studies. STDP depends on the precise temporal relationship of pre- and post-synaptic spikes. Many authors have indicated that a precise balance between long-term potentiation (LTP) and long-term depression (LTD) of STDP is significant for a stable learning. However, a situation in which the balance is maintained precisely is inconceivable in the brain... Downloads: 1 | |

| Theory of Recurrent Neural Network with Common Synaptic Inputs - Masaki Kawamura We discuss the effects of common synaptic inputs in a recurrent neural network. Because of the effects of these common synaptic inputs, the correlation between neural inputs cannot be ignored, and thus the network exhibits sample dependence. Networks of this type do not have well-defined thermodynamic limits, and self-averaging breaks down. We therefore need to develop a suitable theory without relying on these common properties... Downloads: 4 | |

| Mean Field Analysis of Stochastic Neural Network Models with Synaptic Depression - Yasuhiko Igarashi We investigated the effects of synaptic depression on the macroscopic behavior of stochastic neural networks. Dynamical mean field equations were derived for such networks by taking the average of two stochastic variables: a firing state variable and a synaptic variable. In these equations, their average product is decoupled as the product of averaged them because the two stochastic variables are independent... Downloads: 1 | |

| Bifurcation analysis in an associative memory model - Masaki Kawamura We previously reported the chaos induced by the frustration of interaction in a non-monotonic sequential associative memory model, and showed the chaotic behaviors at absolute zero. We have now analyzed bifurcation in a stochastic system, namely a finite-temperature model of the non-monotonic sequential associative memory model. We derived order-parameter equations from the stochastic microscopic equations... Downloads: 2 | |

| Belief Propagation for Error Correcting Codes and Lossy Compression Using Multilayer Perceptrons - Kazushi Mimura The belief propagation (BP) based algorithm is investigated as a potential decoder for both of error correcting codes and lossy compression, which are based on non-monotonic tree-like multilayer perceptron encoders. We discuss that whether the BP can give practical algorithms or not in these schemes. The BP implementations in those kind of fully connected networks unfortunately shows strong limitation, while the theoretical results seems a bit promising... Downloads: 1 | |

| Statistical mechanics of digital halftoning - Jun-ichi Inoue We consider the problem of digital halftoning from the view point of statistical mechanics. The digital halftoning is a sort of image processing, namely, representing each grayscale in terms of black and white binary dots. The digital halftoning is achieved by making use of the threshold mask, namely, for each pixel, the halftoned binary pixel is determined as black if the original grayscale pixel is greater than or equal to the mask value and is determined as white vice versa... Downloads: 1 | |

| The path-integral analysis of an associative memory model storing an infinite number of finite limit cycles - Kazushi Mimura It is shown that an exact solution of the transient dynamics of an associative memory model storing an infinite number of limit cycles with l finite steps by means of the path-integral analysis. Assuming the Maxwell construction ansatz, we have succeeded in deriving the stationary state equations of the order parameters from the macroscopic recursive equations with respect to the finite-step sequence processing model which has retarded self-interactions... Downloads: 1 | |

| On-Line Learning Theory of Soft Committee Machines with Correlated Hidden Units - Steepest Gradient Descent and Natural Gradient Descent - - Masato Inoue The permutation symmetry of the hidden units in multilayer perceptrons causes the saddle structure and plateaus of the learning dynamics in gradient learning methods. The correlation of the weight vectors of hidden units in a teacher network is thought to affect this saddle structure, resulting in a prolonged learning time, but this mechanism is still unclear. In this paper, we discuss it with regard to soft committee machines and on-line learning using statistical mechanics... Downloads: 1 | |

| Naive mean field approximation for image restoration - Hayaru Shouno We attempt image restoration in the framework of the Baysian inference. Recently, it has been shown that under a certain criterion the MAP (Maximum A Posterior) estimate, which corresponds to the minimization of energy, can be outperformed by the MPM (Maximizer of the Posterior Marginals) estimate, which is equivalent to a finite-temperature decoding method. Since a lot of computational time is needed for the MPM estimate to calculate the thermal averages, the mean field method, which is a deter... Downloads: 2 | |

| Acceleration effect of coupled oscillator systems - Toru Aonishi We have developed a curved isochron clock (CIC) by modifying the radial isochron clock to provide a clean example of the acceleration (deceleration) effect. By analyzing a two-body system of coupled CICs, we determined that an unbalanced mutual interaction caused by curved isochron sets is the minimum mechanism needed for generating the acceleration (deceleration) effect in coupled oscillator systems... Downloads: 2 | |

| Statistical Mechanics of Nonlinear On-line Learning for Ensemble Teachers - Hideto Utsumi We analyze the generalization performance of a student in a model composed of nonlinear perceptrons: a true teacher, ensemble teachers, and the student. We calculate the generalization error of the student analytically or numerically using statistical mechanics in the framework of on-line learning. We treat two well-known learning rules: Hebbian learning and perceptron learning. As a result, it is proven that the nonlinear model shows qualitatively different behaviors from the linear model... Downloads: 1 | |

| Analysis of ensemble learning using simple perceptrons based on online learning theory - Seiji Miyoshi Ensemble learning of $K$ nonlinear perceptrons, which determine their outputs by sign functions, is discussed within the framework of online learning and statistical mechanics. One purpose of statistical learning theory is to theoretically obtain the generalization error. This paper shows that ensemble generalization error can be calculated by using two order parameters, that is, the similarity between a teacher and a student, and the similarity among students... Downloads: 2 | |

| Error correcting code using tree-like multilayer perceptron - Florent Cousseau An error correcting code using a tree-like multilayer perceptron is proposed. An original message $\mbi{s}^0$ is encoded into a codeword $\boldmath{y}_0$ using a tree-like committee machine (committee tree) or a tree-like parity machine (parity tree). Based on these architectures, several schemes featuring monotonic or non-monotonic units are introduced. The codeword $\mbi{y}_0$ is then transmitted via a Binary Asymmetric Channel (BAC) where it is corrupted by noise... Downloads: 2 | |

| Theory of localized synfire chain - Kosuke Hamaguchi Neuron is a noisy information processing unit and conventional view is that information in the cortex is carried on the rate of neurons spike emission. More recent studies on the activity propagation through the homogeneous network have demonstrated that signals can be transmitted with millisecond fidelity; this model is called the Synfire chain and suggests the possibility of the spatio-temporal coding... | |

| Statistical Mechanics of On-line Learning when a Moving Teacher Goes around an Unlearnable True Teacher - Masahiro Urakami In the framework of on-line learning, a learning machine might move around a teacher due to the differences in structures or output functions between the teacher and the learning machine. In this paper we analyze the generalization performance of a new student supervised by a moving machine. A model composed of a fixed true teacher, a moving teacher, and a student is treated theoretically using statistical mechanics, where the true teacher is a nonmonotonic perceptron and the others are simple p... | |

| Theory of Interaction of Memory Patterns in Layered Associative Networks - Kazuya Ishibashi A synfire chain is a network that can generate repeated spike patterns with millisecond precision. Although synfire chains with only one activity propagation mode have been intensively analyzed with several neuron models, those with several stable propagation modes have not been thoroughly investigated. By using the leaky integrate-and-fire neuron model, we constructed a layered associative network embedded with memory patterns... | |

| Statistical mechanical evaluation of spread spectrum watermarking model with image restoration - Masaki Kawamura In cases in which an original image is blind, a decoding method where both the image and the messages can be estimated simultaneously is desirable. We propose a spread spectrum watermarking model with image restoration based on Bayes estimation. We therefore need to assume some prior probabilities. The probability for estimating the messages is given by the uniform distribution, and the ones for the image are given by the infinite range model and 2D Ising model... | |

| Associative Memory by Recurrent Neural Networks with Delay Elements - Seiji Miyoshi The synapses of real neural systems seem to have delays. Therefore, it is worthwhile to analyze associative memory models with delayed synapses. Thus, a sequential associative memory model with delayed synapses is discussed, where a discrete synchronous updating rule and a correlation learning rule are employed. Its dynamic properties are analyzed by the statistical neurodynamics. In this paper, we first re-derive the Yanai-Kim theory, which involves macrodynamical equations for the dynamics of ... Downloads: 4 | |

| Application of two-parameter dynamical replica theory to retrieval dynamics of associative memory with non-monotonic neurons - Toshiyuki Tanaka The two-parameter dynamical replica theory (2-DRT) is applied to investigate retrieval properties of non-monotonic associative memory, a model which lacks thermodynamic potential functions. 2-DRT reproduces dynamical properties of the model quite well, including the capacity and basin of attraction. Superretrieval state is also discussed in the framework of 2-DRT. The local stability condition of the superretrieval state is given, which provides a better estimate of the region in which superretr... Downloads: 1 | |

| Sparse and Dense Encoding in Layered Associative Network of Spiking Neurons - Kazuya Ishibashi A synfire chain is a simple neural network model which can propagate stable synchronous spikes called a pulse packet and widely researched. However how synfire chains coexist in one network remains to be elucidated. We have studied the activity of a layered associative network of Leaky Integrate-and-Fire neurons in which connection we embed memory patterns by the Hebbian Learning. We analyzed their activity by the Fokker-Planck method... Downloads: 3 | |

| Low-dimensional chaos induced by frustration in a non-monotonic system - Masaki Kawamura We report a novel mechanism for the occurrence of chaos at the macroscopic level induced by the frustration of interaction, namely frustration-induced chaos, in a non-monotonic sequential associative memory model. We succeed in deriving exact macroscopic dynamical equations from the microscopic dynamics in the case of the thermodynamic limit and prove that two order parameters dominate this large-degree-of-freedom system... Downloads: 3 | |

| Neural network model with discrete and continuous information representation - Jun Kitazono An associative memory model and a neural network model with a Mexican-hat type interaction are the two most typical attractor networks used in the artificial neural network models. The associative memory model has discretely distributed fixed-point attractors, and achieves a discrete information representation. On the other hand, a neural network model with a Mexican-hat type interaction uses a line attractor to achieves a continuous information representation, which can be seen in the working m... Downloads: 4 | |

| Statistical Mechanics of Time Domain Ensemble Learning - Seiji Miyoshi Conventional ensemble learning combines students in the space domain. On the other hand, in this paper we combine students in the time domain and call it time domain ensemble learning. In this paper, we analyze the generalization performance of time domain ensemble learning in the framework of online learning using a statistical mechanical method. We treat a model in which both the teacher and the student are linear perceptrons with noises... Downloads: 7 | |

| Analysis of Bidirectional Associative Memory using SCSNA and Statistical Neurodynamics - Hayaru Shouno Bidirectional associative memory (BAM) is a kind of an artificial neural network used to memorize and retrieve heterogeneous pattern pairs. Many efforts have been made to improve BAM from the the viewpoint of computer application, and few theoretical studies have been done. We investigated the theoretical characteristics of BAM using a framework of statistical-mechanical analysis. To investigate the equilibrium state of BAM, we applied self-consistent signal to noise analysis (SCSNA) and obtaine... Downloads: 6 | |

| Solvable model of a phase oscillator network on a circle with infinite-range Mexican-hat-type interaction - Tatsuya Uezu We describe a solvable model of a phase oscillator network on a circle with infinite-range Mexican-hat-type interaction. We derive self-consistent equations of the order parameters and obtain three non-trivial solutions characterized by the rotation number. We also derive relevant characteristics such as the location-dependent distributions of the resultant frequencies of desynchronized oscillators... Downloads: 7 | |

| Synapse efficiency diverges due to synaptic pruning following over-growth - Kazushi Mimura In the development of the brain, it is known that synapses are pruned following over-growth. This pruning following over-growth seems to be a universal phenomenon that occurs in almost all areas -- visual cortex, motor area, association area, and so on. It has been shown numerically that the synapse efficiency is increased by systematic deletion. We discuss the synapse efficiency to evaluate the effect of pruning following over-growth, and analytically show that the synapse efficiency diverges a... Downloads: 7 | |

| Influence of synaptic depression on memory storage capacity - Yosuke Otsubo Synaptic efficacy between neurons is known to change within a short time scale dynamically. Neurophysiological experiments show that high-frequency presynaptic inputs decrease synaptic efficacy between neurons. This phenomenon is called synaptic depression, a short term synaptic plasticity. Many researchers have investigated how the synaptic depression affects the memory storage capacity. However, the noise has not been taken into consideration in their analysis... Downloads: 2 | |

| Bayes-optimal inverse halftoning and statistical mechanics of the Q-Ising model - Yohei Saika On the basis of statistical mechanics of the Q-Ising model, we formulate the Bayesian inference to the problem of inverse halftoning, which is the inverse process of representing gray-scales in images by means of black and white dots. Using Monte Carlo simulations, we investigate statistical properties of the inverse process, especially, we reveal the condition of the Bayes-optimal solution for which the mean-square error takes its minimum... Downloads: 1 | |